UCSG Shallow Parsing: Optimum Chunk Sequence Selection

This paper is about syntactic analysis of natural language sentences. The focus is on wide coverage partial parsing architectures. In this work we enhance and enrich the UCSG shallow parsing architecture being developed here over the last many years. UCSG architecture combines linguistic grammars in the form of Finite State Machines for recognising all potential chunks and HMMs to rate and rank the chunks so produced. Here we have explored Mutual Information statistics for rating and ranking chunks as also complete parses (chunk sequences) so as to place the best parses near the top. The main aim of this work is to identify the best word group (chunk) sequence or global parse for a given sentence using a information-theoretic measure called mutual information. This method is based on the hypothesis that the best chunk can be obtained by analysing the mutual information values of the chunk tag n-grams. In the initial version of UCSG, HMMs were local to chunks. Global information such as the probability of a chunk of a given type starting a sentence or the probability of a chunk of a particular type occurring next to a chunk of a given type are also useful. We try to capture this global information in the form of mutual information score and use it in improving the ranks of correct chunks. Combining the two methods, namely HMMs and Mutual Information, to get the full information of a chunk to improve the ranks further, is another important aspect of this work. Later, a Best First Search module uses these ranked chunks to find the final parse. We have also added 1000 sentence manually parsed corpus to the existing 4000 manually parsed data. Experiments on the existing and newly added text corpora are included.